Assessing the performance of prediction models: a framework for traditional and novel measures.
نویسندگان
چکیده
The performance of prediction models can be assessed using a variety of methods and metrics. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance (or c) statistic for discriminative ability (or area under the receiver operating characteristic [ROC] curve), and goodness-of-fit statistics for calibration.Several new measures have recently been proposed that can be seen as refinements of discrimination measures, including variants of the c statistic for survival, reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI). Moreover, decision-analytic measures have been proposed, including decision curves to plot the net benefit achieved by making decisions based on model predictions.We aimed to define the role of these relatively novel approaches in the evaluation of the performance of prediction models. For illustration, we present a case study of predicting the presence of residual tumor versus benign tissue in patients with testicular cancer (n = 544 for model development, n = 273 for external validation).We suggest that reporting discrimination and calibration will always be important for a prediction model. Decision-analytic measures should be reported if the predictive model is to be used for clinical decisions. Other measures of performance may be warranted in specific applications, such as reclassification metrics to gain insight into the value of adding a novel predictor to an established model.
منابع مشابه
Assessing the Performance of Corroding RC Bridge Decks: A Critical Review of Corrosion Propagation Models
Corrosion of steel reinforcement is one of the most prevalent causes of reinforced concrete (RC) structures deterioration in chloride-contaminated environments. As a result, evaluating the impact of any possible corrosion-induced damages to reinforced concrete bridges strongly affects management decisions: such as inspection, maintenance and repair actions. The corrosion propagation phase is a ...
متن کاملاندازه گیری عملکرد بیمارستان
Introduction: The core requirement of successful and first- in- class organizations is doing right things and doing things right. Recent organizations should be able to have excellent performance from strategic and operational point of view so that they can face the current and future world challenges. Performance measurement is one of the ways for directing organization to the right targets ...
متن کاملA novel risk-based analysis for the production system under epistemic uncertainty
Risk analysis of production system, while the actual and appropriate data is not available, will cause wrong system parameters prediction and wrong decision making. In uncertainty condition, there are no appropriate measures for decision making. In epistemic uncertainty, we are confronted by the lack of data. Therefore, in calculating the system risk, we encounter vagueness that we have to use ...
متن کاملA Novel Fuzzy-Genetic Differential Evolutionary Algorithm for Optimization of A Fuzzy Expert Systems Applied to Heart Disease Prediction
This study presents a novel intelligent Fuzzy Genetic Differential Evolutionary model for the optimization of a fuzzy expert system applied to heart disease prediction in order to reduce the risk of heart disease. To this end, a fuzzy expert system has been proposed for the prediction of heart disease. The proposed model can be used as a tool to assist physicians. In order to: (1) tune the para...
متن کاملOptimal Portfolio Allocation based on two Novel Risk Measures and Genetic Algorithm
The problem of optimal portfolio selection has attracted a great attention in the finance and optimization field. The future stock price should be predicted in an acceptable precision, and a suitable model and criterion for risk and the expected return of the stock portfolio should be proposed in order to solve the optimization problem. In this paper, two new criterions for the risk of stock pr...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- Epidemiology
دوره 21 1 شماره
صفحات -
تاریخ انتشار 2010